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Abstract 

Deaf-blind people face major communication issues due to a lack of appropriate technology. Our system, a 
braille device designed specifically for the deaf-blind, provides a solution to the problem by allowing users to 
feel and interpret each letter by touch in real-time, enabling smooth conversation. They can understand 
messages and interact meaningfully with this tactile interface, thereby overcoming the communication gap. 
The innovation allows deaf-blind people to participate more completely in social interactions and access 
information independently, improving their quality of life and promoting inclusion. 

Keywords: Assistive technology; Clean technology; Deaf-blind; Refreshable braille display; Speech 


recognition. 


1. Introduction 

Deaf-blind people often rely on tactile methods such 
as braille, but not everyone knows this tactile 
language. Therefore, deaf-blind people often find it 
difficult to interact with them. Here comes the 
relevance of this system. By offering real-time braille 
translations in three modes: audio, image and text, the 
proposed system seeks to bridge the communication 
gap between deaf-blind and abled people. With an 
emphasis on device affordability and accessibility, 
the paper intends to improve participation of the deaf- 
blind in social, educational, and _ professional 
environments. The goal is to develop a socialization 
aid that will let the deaf-blind people communicate in 
real-time using image captioning, text-to-braille, and 
speech-to-text algorithms. Overall, the device seeks 
to increase participation, close communication gaps, 
encourage inclusivity, and provide a_ flexible 
communication platform for those with dual sensory 
impairments. 

2. Literature Review 

The advances in assistive technology for the deaf- 
blind are examined in this literature review, with 
particular attention paid to text-to-braille, speech-to- 
text, braille display devices, and image captioning 
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techniques. It examines the limitations of current 
technologies and explores how machine learning and 
artificial intelligence can be used to improve existing 
technologies. Integrating touch-sensitive gloves with 
two-way communication is a promising wearable 
technology approach [1]. The HC-05 Bluetooth 
module and 16-bit ultra-low-power Microcontroller 
Unit (MCU) enable this. However, the complex 
design and operation may make it difficult for users 
to adapt to the technology. Addressing these input 
and complexity issues are necessary. In a 
communication system, vibration buttons are used to 
display the braille output [2]. It uses software to 
decode braille code and uses vibration patterns to 
transmit information. However, it has its drawbacks, 
such as difficulty in tactile feedback and a limited 
input format. Additionally, the message must be 
understood by the deaf-blind individual, making real- 
time communication difficult and requiring 
additional training or adjustments. A _ speaker- 
independent system using the HM2007 integrated 
circuit employs Hidden Markov Models (HMM) as 
its acoustic model and represents a_ significant 
breakthrough in speech recognition [3]. Particularly 
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fascinating is its integration with solenoid actuators, 
which enables braille output display and provides 
vision-challenged individuals with an inclusive mode 
of communication. Despite these advantages, a 
drawback arises from the fact that pop-up pins have a 
tactile responsiveness that solenoid actuator output 
lacks. Because braille cannot be felt by the hand as 
easily, its effectiveness and user experience are 
limited. The device’s sole reliance on speech as an 
input method may limit their potential for integrating 
alternative methods. There are three image 
captioning methods: CNN-RNN, CNN-CNN, and 
reinforcement-learning frameworks [4]. CNN-RNN 
uses Recurrent Neural Network (RNN) for sequential 
language production and Convolutional Neural 
Network (CNN) for image feature extraction while 
reinforcement-based frameworks use reinforcement 
learning techniques to optimize photo captioning. 
The significance of image captioning approaches in 
computer vision and natural language processing 
calls for more research and comparison. The primary 
conversion technologies for assistive technology 
systems include text-to-speech (TTS), speech-to-text 
(STT), braille-to-text (BTT), and _text-to-braille 
(TTB) [5]. Every technique has benefits and 
drawbacks of its own. However, the current state of 
spoken language technology is inadequate compared 
to human speech output. Research and development 
in this constantly changing field are crucial for blind 
and visually impaired individuals. A report on 
speech-to-braille language encoding for blind people 
[6] presents an innovative mobile communication 
system featuring an HC-05 Bluetooth module, an 
Liquid-Crystal Display (LCD) for braille output, an 
ATmegal6 Microcontroller for processing, and a 
mobile app for communication. However, because 
the system only accepts audio as input and does not 
provide tactile output, it is not beneficial to users who 
are deaf-blind [7]. The literature review offers a 
comprehensive overview of technologies for sensory 
impairments, focusing on the blind and visually 
impaired. Studied approaches include touch-sensitive 
gloves, vibration buttons, and various conversion 
technologies [8]. Common limitations involve 
complex design, requirement of additional training, 
limited input formats, and real-time communication 
challenges. The proposed system addresses these by 
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combining simple tactile feedback, intuitive 
interaction, and versatile input methods [9]. To the 
best of our knowledge, a system similar to the 
proposed system does not currently exist. The 
contributions of this work are as follows: It has an 
innovative approach for addressing the challenges 
faced by the deaf-blind community through a 
comprehensive system that includes three modes of 
input [10]. This system tackles the limitations of 
previous systems by _ offering _ real-time 
communication through a combination of speech 
recognition, image captioning, and text-to-braille 
mapping techniques. Unlike existing systems that 
may not fully cater to the needs of the deaf-blind, this 
proposed system focuses on empowering deaf-blind 
individuals by providing a versatile and interactive 
communication platform. Additionally, the system 
addresses the challenges of device ownership and 
cost that have limited the usability of previous 
systems by making a cost-efficient and compact 
design. The remaining sections of this paper are 
structured as follows: Section 3 introduces the 
proposed system, Section 4 covers the methodologies 
employed, and Section 5 delves into the system's 
implementation and potential challenges during 
development. Following this, Section 6 contains 
intended results and discussion about the system’s 
impacts, while Section 7 provides a comprehensive 
conclusion about the system, including insights into 
future developments. 

3. Proposed System 

In response to the evolving landscape of assistive 
technology, this paper proposes a novel and 
comprehensive system designed to address the 
challenges faced by deaf-blind people. The proposed 
system is composed of both software and hardware 
components. The software component consists of a 
mobile application that the abled person can use to 
initiate the conversation. The message transfer 
between an abled person and a deaf-blind person is 
handled by the software component. The hardware is 
an electronic braille system designed to aid the deaf- 
blind in perceiving the message being transmitted. It 
is composed of Arduino, Bluetooth module, 
MOLBED (Modular Low-Cost Braille Electronic 
Display), and other parts. A single cell of MOLBED 
is shown in Figure 1. 
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Figure 1 MOLBED 


The key innovation of the proposed system lies in 
accepting three input types: text, audio, and image. 
Through the smartphone application, an abled person 
can send a text, audio, or image message. By offering 
this feature, this system differs greatly from other 
alternatives. The proposed system presents several 
advantages, including the facilitation of real-time 
communication between deaf-blind individuals and 
abled individuals. The electronic braille device 
utilized for displaying transmitted messages is 
refreshable, ensuring timely interaction and also 
generates no waste materials. The conventional 
method for reading braille involves printing braille 
characters on paper, resulting in the disposal of these 
papers after use, leading to significant paper wastage. 
The proposed system addresses this challenge by 
using a refreshable braille display so that the braille 
can be printed again and again on the same device. 
This advancement significantly reduces paper 
wastage, offering a more sustainable approach to 
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braille reading. The mobile application offers a 
versatile communication experience with three 
modes: text, audio, and image inputs. The specially 
developed device for the deaf-blind is not only 
portable but also features a simple design for easy 
use. Additionally, the mobile application has a user- 
friendly interface, allowing individuals with sight and 
hearing abilities to effortlessly engage, thus 
significantly improving communication between 
deaf-blind individuals and others. The system 
described in this paper significantly contributes to 
improving social engagement within the deaf-blind 
community by addressing communication delays. In 
situations where the abled person is unable to speak, 
the system facilitates seamless interaction through 


text-based communication. Moreover, to 
accommodate individuals with limited English 
proficiency, the system incorporates image 


captioning techniques, allowing users to capture 
images that are then converted into corresponding 
braille. These inclusive features empower the deaf- 
blind to communicate effectively, irrespective of the 
abled person's language proficiency or ability to 
speak, fostering a sense of inclusivity and enhancing 
social engagement for the deaf-blind community. 

4. Methodology 

To facilitate communication and accessibility for the 
deaf-blind community, the proposed system 
incorporates multiple techniques. Figure 2 shows the 
process flowchart for the system. 


Bluetooth Module 


Braille Display Device 


Figure 2 Process Flowchart 
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The user has three options for input: text, image, and 
audio. The speech and image inputs are converted to 
text and subsequently to braille while the text input is 
directly converted into braille. A Bluetooth module is 
used to send the converted data to a braille device, 
which then displays the final output. The various 
technologies adopted in the system are described 
below. 

4.1.Speech Recognition 
Speech recognition, also known as speech-to-text, is 
the ability of a machine or a program to identify 
words spoken aloud and convert them into readable 
text. This technology uses various algorithms and 
computation techniques, including deep neural 
networks and recurrent neural networks, to convert 
spoken language into written language. It enables the 
recognition and translation of spoken language into 
text through computational linguistics. This allows 
deaf-blind individuals to participate in conversations 
by converting their speech into text that can be 
displayed on the braille display. Therefore, speech 
recognition is a crucial aspect of this system. Here, 
the proposed system will be using the Speech-to-Text 
plugin of Flutter, which is both easy to use and 
provides accurate results. 

4.2.Image Captioning 
It contains techniques for automatically generating 
text descriptions of images, enabling deaf-blind 
individuals to understand the visual content during 
conversations. Techniques such as deep learning and 
computer vision can be used to automatically 
generate text descriptions of images. This 
technology, when integrated into the system, will 
enable deaf-blind users to better understand and 
appreciate visual content, thereby improving their 
overall communication experience. The system will 
be using deep learning-based image captioning 
models to achieve image-to-text conversion. 

4.3.Text-to-Braille Mappings 
Text-to-braille mappings involve techniques for 
converting text input into braille characters, allowing 
deaf-blind individuals to read and write text on a 
braille display. Techniques such as_ character 
recognition and segmentation algorithms can be 
employed to convert text input into braille characters. 
This technology will enable deaf-blind users to 
navigate and interact with text-based information 
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more efficiently, thereby improving their overall 
communication and accessibility experience. Here, in 
this system, a simple mapping is implemented using 
a Python dictionary for the text-to-braille conversion 
4.4.Braille Device 

The Arduino, Bluetooth module, and MOLBED are 
the hardware components of the braille device. These 
elements are responsible for the hardware's core 
operations. The system employs an Arduino to 
oversee and manage MOLBED’s | functions. 
Transmission of braille characters from the mobile 
application to the hardware display is facilitated by a 
Bluetooth module. The MOLBED features pins for 
displaying messages from a mobile application. Each 
braille character on the prototype is represented by a 
set of six pins, crafted with various components, 
including 3D-printed parts, magnets, coil, etc. 

5. Implementation and Challenges 

The implementation of the proposed system consists 
of three phases: software, hardware, and integration. 
The front-end application that receives user input and 
sends the braille to the hardware device is 
implemented during the software phase. It consists of 
different pages that each of the three input types 
requires. The entire application is written in Dart, 
utilizing the Flutter framework. The application's 
home page consists of three options: 'Audio’, Image’, 
and ‘Text’. Each of the options leads to the 
corresponding page that takes in the input and sends 
it to the backend for processing. Python is used in the 
backend implementation to take advantage of its 
extensive machine learning libraries for image 
captioning, speech-to-text, and other uses. The 
backend includes text-to-braille conversion methods, 
image captioning techniques based on deep learning 
models, speech-to-text, and finally braille-binary 
mappings. The braille-binary mapping is achieved 
using dictionaries. In the end, the backend will 
transmit the binary to the hardware device over 
Bluetooth. Therefore, the Bluetooth module acts as 
the middleware for the transmission. The actual 
braille display device is implemented in the hardware 
phase while in the integration phase, the software and 
the hardware components are combined to produce 
the final system. The hardware device receives the 
braille input from the front-end application through 
the Bluetooth module and uses the MOLBED system 
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to display it. During the development of this system, 
several challenges may arise. These challenges can be 
categorized into different aspects: 

5.1.Technical Challenges 
The system relies on advanced technologies such as 
speech recognition, image captioning, and text-to- 
braille mappings. Developing and integrating these 
technologies into the system may pose challenges, as 
they must be robust, user-friendly, and accessible for 
individuals with vision and hearing impairments. 
Also, these conversions need to work in real time and 
be fast and accurate. 

5.2.Hardware and Software Integration 
The system requires seamless integration of hardware 
and software components to work properly. Ensuring 
compatibility and smooth communication between 
these components may present challenges, as 
different technologies and platforms need to work 
together effectively. 

5.3.Accessibility and Affordability 
The proposed system seeks to make communication 
more affordable and accessible for individuals with 
vision and hearing impairments. Ensuring that the 
system is affordable and accessible to a wide range of 
users may pose challenges, as it needs to consider 
factors such as device ownership, cost of assistive 
devices, and user preferences. In conclusion, the 
development phase will be facing various challenges, 
and addressing these challenges will be crucial for the 
successful implementation and adoption of the 
system by individuals with vision and hearing 
impairments. 
6. Results and Discussion 
The proposed system will yield significant results in 
the field of assistive technology, particularly for the 
deaf-blind community. To improve social inclusion, 
communication, and accessibility for the deaf-blind 
community, the system's outcomes cover both its 
technical and social effects. To begin with, the system 
features a mobile application that is easy to use and 
fully functional, enabling real-time communication 
and information access. Three input options are 
displayed on the application's home page: text, audio, 
and image, as Figure 3 illustrates. All these options 
lead to the relevant pages. 
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Figure 3 Home Page of the Application 


Figure 4 illustrates the resulting braille output of an 
audio input, represented in binary as 0's and 1's. The 
received input, in the form of both audio and images, 
undergoes a two-step conversion process to get the 
final braille output. For textual input, a direct 
conversion to braille is performed. 


Figure 4 Output of Speech-to-Text and Text-to- 
Binary Mappings 


The mobile application for the system is compatible 
with widely available technologies, such as Android 
platforms, and offers a versatile communication 
experience for both able and deaf-blind users. Figure 
5 depicts the model of a hardware device that displays 
braille characters in real-time and facilitates smooth 
communication for people with vision and hearing 
impairments and Figure 6 illustrates the prototype for 
a single cell of the device. A tactile braille display that 
efficiently converts several input modalities, such as 
text, audio, and image, into braille output is the 
intended outcome. This hardware component is user- 
friendly, portable, and cost-effective, addressing the 
unique communication needs of the target 
demographic. 
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Figure 5 Braille Display Device 


Figure 6 Prototype for A Single Cell of the Device 


The proposed system aims to foster greater 
inclusivity and participation for individuals with dual 
sensory impairments, encouraging them to engage in 
social, educational, and professional environments. 
By enabling real-time communication and 
information access, the system seeks to empower 
deaf-blind individuals, thereby enhancing their 
overall quality of life. The paper aims to address the 
accessibility and affordability challenges faced by 
individuals with vision and hearing impairments. The 
final outcome is a user-friendly and cost-effective 
communication system that caters to the diverse 
needs of the deaf-blind community that promotes 
greater independence and social engagement. The 
proposed system is expected to have a positive impact 
on the lives of deaf-blind individuals by providing 
them with a means to participate more fully in 
society. 

Conclusion and Future Scope 

In conclusion, this paper proposes a novel solution to 
address the social isolation experienced by deaf-blind 
individuals. The proposed system uses tactile 
feedback to enable communication and _ social 
interaction, allowing deaf-blind individuals to 
connect with others and participate in social 
activities. The system has the potential to improve the 


International Research Journal on Advanced Engineering Hub (IRJAEH) 


International Research Journal on Advanced Engineering Hub (IRJAEH) 


e ISSN: 2584-2137 
Vol. 02 Issue: 06 June 2024 
Page No: 1825- 1831 


https://irjaeh.com 
https://doi.org/10.47392/IRJAEH.2024.0251 


quality of life for deaf-blind individuals and promote 
their inclusion in society. The future vision for the 
system is to enhance communication in the deaf-blind 
community by enabling two-way communication and 
initiating dynamic conversations. Other future 
developments include reinforcing hardware with a 
robust battery backup, ensuring — sustained 
functionality. Additionally, the innovative software 
of the system can be implemented on wearable 
devices like smartwatches, aiming for broader 
accessibility and seamless integration into users' 
daily lives. This future scope aims for continuous 
improvement, adaptability, and breaking down 
communication barriers for the deaf-blind. 
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